Visual Haptic Reasoning: Estimating Contact Forces by Observing Deformable Object Interactions

Robotic manipulation of highly deformable cloth presents a promising opportunity to assist people with several daily tasks, such as washing dishes; folding laundry; or dressing, bathing, and hygiene assistance for individuals severe motor impairments. In this letter, we introduce formulation that enables collaborative robot perform visual haptic reasoning cloth—the act inferring the location magnitude applied forces during physical interaction. We present two distinct model representations, trained in physics simulation, enable using only kinematic observations. conducted quantitative evaluations these models simulation robot-assisted dish demonstrate can generalize across different tasks varying interactions, human body sizes, object shapes. also results real-world mobile manipulator, which used our simulation-trained estimate contact while performing physically assistive cloth. Videos be found at project webpage.